EMU: A Web Portal Generator

A web portal is a set of HTML documents that comprise a tree rooted at the portal entry site; the leaves are links to other websites. Each node within the tree represents a cluster of documents, decreasing in generality as one descends down the tree. Portals are useful for organizing and enabling users to search large repositories of documents, either centralized or distributed. Whereas search engines require the user to specify search criteria, typically as key words, to search an opaque document bank, a portal visualizes the document bank and can be particularly useful if the query is not amenable to representation by key words. Popular portals, such as those hosted by AltaVista, Google, Yahoo, and dmoz.org, are created by hand. For example, the Open Directory Project at dmoz.org enlists volunteers to manages subtopics. In contrast, EMU is an automated system that uses statistical NLP and unsupervised machine learning techniques to generate a web portal. While the quality of the portal must certainly suffer, less frequented document repositories and Usenets may benefit from an automated system. Additionally, EMU is practical for generating daily navigators of newspapers, internal documents of a company, and other dynamic document banks. Many companies, for example, provide employees with daily or weekly summaries of domain-specific periodicals relating to the company’s business; EMU would provide a practical alternative. Finally, hierarchical clustering can serve as a preprocessing step for fast online clustering used in document retrieval and user queries. Previous work has used hierarchical clustering for navigational purposes. Scatter/Gather [3] uses a clustering approach to summarize a set of documents; however, they adapt an interactive approach with the user. A typical session consists of (flat) clustering a set of documents and labeling the clusters, allowing the chooser to select a cluster to explore, and repeating the process on the new cluster. This corresponds directly to the EMU interface of selecting superclusters to explore in more depth, although the interactive approach allows the system to take advantage of fast non-hierarchical clustering algorithms. EMU, however, runs only once to generate a summary; hence, it is appropriate for generating summaries that many people will read via the web. This report is organized as follows. In Section 2, we present an overview of the EMU system, while in Section 3, we consider EMU’s document representation and learning algorithms. Section 4 presents our testing techniques; we also evaluate our results on a test set. Finally, Section 5 concludes with ideas for applications and future work.